Head mounted display

ABSTRACT

A head mounted display is used in a state of being mounted on a user&#39;s head and includes a convex lens disposed at a position facing the user&#39;s cornea when the head mounted display is mounted. An infrared light source emits infrared light toward the convex lens. A camera captures an image including the user&#39;s cornea in a subject. A housing houses the convex lens, the infrared light source, and the camera. The convex lens is provided with a plurality of reflection regions that reflects infrared light in an inside of the convex lens. The infrared light source causes a pattern of infrared light to appear on the user&#39;s cornea by emitting infrared light to each of the plurality of reflection regions provided in the convex lens.

TECHNICAL FIELD

This disclosure relates to a head mounted display.

BACKGROUND ART

A technique is known in which the eyesight direction of a user isdetected by emitting non-visible light such as near-infrared light tothe user's eyes, and analyzing an image of the user's eyes includingreflected light. Information of the detected eyesight direction of theuser is reflected on the monitor of, for example, a PC (PersonalComputer), a game console or the like, and thus use as a pointing devicehas been realized.

CITATION LIST Patent Literature [Patent Literature 1]

Japanese Unexamined Patent Application, First Publication No. H2-264632

SUMMARY OF INVENTION Technical Problem

A head mounted display is an image display device that presents athree-dimensional image to a user wearing the device. Generally, thehead mounted display is used in a state of being mounted to cover thevisual range of a user. For this reason, a user wearing the head mounteddisplay has an external image shielded. When the head mounted display isused as a display device of an image of a moving picture, a game or thelike, it is difficult for a user to visually recognize an input devicesuch as a controller.

Therefore, the usability of a head mounted display as a substitute for apointing device by detecting the eyesight direction of a user wearingthe display is of convenience. Particularly, the acquisition ofgeometric information (information of spatial coordinates or a shape) ofa user's cornea in a state where the user wears a head mounted displayis useful in estimating the eyesight direction of the user.

It could therefore be helpful to provide a technique of detectinggeometric information of the cornea of a user wearing a head mounteddisplay.

Solution to Problem

Provided is a line-of-sight detection system including a head-mounteddisplay and a line-of-sight detection device, the head-mounted displayincludes an image display element that includes a plurality of pixels,each pixel including sub-pixels that emit red, green, blue, invisiblelight, and displays an image to be viewed by a user; an imaging unitthat images an eye of the user wearing the head-mounted display on thebasis of the invisible light emitted from the sub-pixel that emits theinvisible light; and a transmission unit that transmits a captured imagecaptured by the imaging unit, and the line-of-sight detection deviceincludes a reception unit that receives the captured image; and aline-of-sight detection unit that detects a line of sight of the eye ofthe user based on the captured image.

Further, in order to resolve the problem, a head-mounted displayaccording to an aspect of the present invention is a head-mounteddisplay mounted on a head of a user and used, and includes a convex lensdisposed in a position facing a cornea of the user when the head-mounteddisplay is mounted; an image display element that includes a pluralityof pixels, each pixel including sub-pixels that emit red, green, blue,invisible light, and displays an image to be viewed by a user; a camerathat images a video including the cornea of the user as a subject; and ahousing that houses the convex lens, the image display element, and thecamera.

Further, in the head-mounted display, the head-mounted display mayfurther include a control unit that selects the pixel that emitsinvisible light among the plurality of pixels constituting the imagedisplay element and causes the selected pixel to emit light.

Further, the control unit may change the pixel that emits the invisiblelight when a predetermined time elapses.

Further, the control unit may switch and control a light emission timingof the sub-pixel that emit the invisible light and the sub-pixel otherthan the sub-pixels that emit the invisible light.

Meanwhile, any combination of the aforementioned components, andimplementation of our displays in the form of methods, devices, systems,computer programs, data structures, recording mediums, and the like maybe considered part of this disclosure.

It is thus possible to provide a technique of detecting geometricinformation of the cornea of a user wearing a head mounted display.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram schematically illustrating a general view of anexample of our image system.

FIG. 2 is a diagram schematically illustrating an optical configurationof an image display system housed by a housing.

FIGS. 3(a) and 3(b) are schematic diagrams illustrating a reflectionregion.

FIGS. 4(a) and 4(b) are diagrams schematically illustrating an exampleof dot patterns generated by a reflection region of a convex lens.

FIG. 5 is a schematic diagram illustrating a relationship between acaptured dot pattern and a structure of a subject.

FIG. 6 is a diagram schematically illustrating a functionalconfiguration of an image reproducing device.

FIG. 7 is a schematic diagram illustrating an eyesight direction of auser.

FIG. 8 is a schematic diagram illustrating calibration in an eyesightdirection which is executed by a head mounted display.

FIG. 9 is a diagram illustrating an example of a pixel configuration ofan image display element.

FIG. 10 is a diagram illustrating an example in which an eye of a useris directly irradiated with image light, which includes infrared light.

FIG. 11A and FIG. 11B illustrate an example in which a line-of-sightdetection is performed on the basis of an image reflected in the eye ofthe user.

DESCRIPTION OF EMBODIMENTS

FIG. 1 is a diagram schematically illustrating a general view of animage system 1 according to an example. The image system 1 includes ahead mounted display 100 and an image reproducing device 200. As shownin FIG. 1, the head mounted display 100 is used in a state of beingmounted on the head of a user 300.

The image reproducing device 200 generates an image displayed by thehead mounted display 100. Although not limited, as an example, the imagereproducing device 200 is a device capable of reproducing an image of astationary game console, a portable game console, a PC, a tablet, asmartphone, a phablet, a video player, a television or the like. Theimage reproducing device 200 connects to the head mounted display 100 ina wireless or wired manner. In an example shown in FIG. 1, the imagereproducing device 200 is wirelessly connected to the head mounteddisplay 100. The wireless connection of the image reproducing device 200to the head mounted display 100 can be realized using, for example, awireless communication technique such as known WI-FI (RegisteredTrademark) or BLUETOOTH (Registered Trademark). Although not limited, asan example, transmission of an image between the head mounted display100 and the image reproducing device 200 is executed according to thestandard of MIRACAST (Trademark), WIGIG (Trademark), WHDI (Trademark),or the like.

Meanwhile, FIG. 1 illustrates an example when the head mounted display100 and the image reproducing device 200 are different devices. However,the image reproducing device 200 may be built into the head mounteddisplay 100.

The head mounted display 100 includes a housing 150, a mounting fixture160, and a headphone 170. The housing 150 houses an image display systemsuch as an image display element that presents an image to the user 300,or a wireless transmission module such as a WI-FI module or a BLUETOOTH(Registered Trademark) module which is not shown. The mounting fixture160 mounts the head mounted display 100 on the head of the user 300. Themounting fixture 160 can be realized by, for example, a belt, an elasticband or the like. When the user 300 mounts the head mounted display 100using the mounting fixture 160, the housing 150 is disposed at aposition where the eyes of the user 300 are covered. For this reason,when the user 300 mounts the head mounted display 100, the visual rangeof the user 300 is shielded by the housing 150.

The headphone 170 outputs a voice of an image reproduced by the imagereproducing device 200. The headphone 170 may be fixed to the headmounted display 100. Even in a state where the user 300 mounts the headmounted display 100 using the mounting fixture 160, the user can freelyattach and detach the headphone 170.

FIG. 2 is a diagram schematically illustrating an optical configurationof an image display system 130 housed by the housing 150. The imagedisplay system 130 includes a near-infrared light source 103, an imagedisplay element 108, a hot mirror 112, a convex lens 114, a camera 116,and an image output unit 118.

The near-infrared light source 103 is a light source capable of emittinglight of a near-infrared (approximately 700 nm to 2,500 nm) wavelengthband. The near-infrared light is light of a wavelength band ofnon-visible light which is not able to be generally observed with anaked eye of the user 300.

The image display element 108 displays an image for presentation to theuser 300. The image displayed by the image display element 108 isgenerated by a GPU (Graphic Processing Unit), not shown, within theimage reproducing device 200. The image display element 108 can berealized using, for example, a known LCD (Liquid Crystal Display), anorganic EL display (Organic Electro-Luminescent Display) or the like.

When the user 300 mounts the head mounted display 100, the hot mirror112 is disposed between the image display element 108 and the cornea 302of the user 300. The hot mirror 112 has a property of transmittingvisible light generated by the image display element 108, but reflectingnear-infrared light.

The convex lens 114 is disposed on the opposite side to the imagedisplay element 108 with respect to the hot mirror 112. In other words,when the user 300 mounts the head mounted display 100, the convex lens114 is disposed between the hot mirror 112 and the cornea 302 of theuser 300. That is, when the head mounted display 100 is mounted to theuser 300, the convex lens 114 is disposed at a position facing thecornea 302 of the user 300.

The convex lens 114 condenses image display light that passes throughthe hot mirror 112. For this reason, the convex lens 114 functions as animage enlargement unit that enlarges an image generated by the imagedisplay element 108 and presents the enlarged image to the user 300.Meanwhile, for convenience of description, only one convex lens 114 isshown in FIG. 2, but the convex lens 114 may be a lens group configuredby combining various lenses, and may be configured such that one lenshas a curvature and the other lens is a planar one-sided convex lens.

The near-infrared light source 103 is disposed at the lateral side ofthe convex lens 114. The near-infrared light source 103 emits infraredlight toward the inside of the convex lens 114. The convex lens 114 isprovided with a plurality of reflection regions that reflect theinfrared light inside the lens. These reflection regions can be realizedby providing fine regions having different refractive indexes in theinside of the convex lens 114. Meanwhile, providing the regions havingdifferent refractive indexes in the convex lens 114 can be realizedusing a known laser machining technique. The reflection region isprovided at a plurality of places in the inside of the convex lens 114.

Near-infrared light emitted toward the inside of the convex lens 114 bythe near-infrared light source 103 is reflected from the reflectionregion inside the convex lens 114 and directed to the cornea 302 of theuser 300. Meanwhile, since the near-infrared light is non-visible light,the user 300 is almost not able to perceive the near-infrared lightreflected from the reflection region. In addition, the reflection regionis a region which is as large as a pixel constituting the image displayelement 108 or is finer. For this reason, the user 300 is almost notable to perceive the reflection region, and is able to observe imagelight emitted by the image display element 108. Meanwhile, the detailsof the reflection region will be described later.

Although not shown, the image display system 130 of the head mounteddisplay 100 includes two image display elements 108, and can generate animage for presentation to the right eye of the user 300 and an image forpresentation to the left eye independently of each other. For thisreason, the head mounted display 100 can present a parallax image forthe right eye and a parallax image for the left eye, respectively, tothe right eye and the left eye of the user 300. Thereby, the headmounted display 100 can present a stereoscopic image having a sense ofdepth to the user 300.

As described above, the hot mirror 112 transmits visible light, andreflects near-infrared light. Therefore, image light emitted by theimage display element 108 passes through the hot mirror 112 and reachesthe cornea 302 of the user 300. In addition, infrared light emitted fromthe near-infrared light source 103 and reflected from the reflectionregion inside the convex lens 114 reaches the cornea 302 of the user300.

The infrared light reaching the cornea 302 of the user 300 is reflectedfrom the cornea 302 of the user 300, and directed to the direction ofthe convex lens 114 again. This infrared light passes through the convexlens 114, and is reflected from the hot mirror 112. The camera 116includes a filter that shields visible light, and captures near-infraredlight reflected from the hot mirror 112. That is, the camera 116 is anear-infrared camera that captures near-infrared light emitted from thenear-infrared light source 103 and reflected from the cornea of the user300.

The image output unit 118 outputs an image captured by the camera 116 toan eyesight detection unit that detects the eyesight direction of theuser 300. The image output unit 118 also outputs the image captured bythe camera 116 to a cornea coordinate acquisition unit that acquiresspatial coordinates of the user's cornea. Specifically, the image outputunit 118 transmits the image captured by the camera 116 to the imagereproducing device 200. The eyesight detection unit and the corneacoordinate acquisition unit will be described later, but can be realizedby an eyesight detecting program and a cornea coordinate acquiringprogram executed by a CPU (Central Processing Unit) of the imagereproducing device 200. Meanwhile, when the head mounted display 100 hasa computing resource of a CPU, a memory or the like, the CPU of the headmounted display 100 may execute a program to operate the eyesightdetection unit.

Although a detailed description will be given later, in the imagecaptured by the camera 116, a bright point of the near-infrared lightreflected by the cornea 302 of the user 300 and an image of the eyeincluding the cornea 302 of the user 300 observed at a near-infraredwavelength band are captured.

In the convex lens 114, a plurality of reflection regions are formed sothat a pattern of infrared light appearing on the cornea 302 of the user300 forms structured light. The “structured light” refers to light usedin one method of three-dimensional measurement of an object called astructured light method. More specifically, the structured light islight emitted to cause a light pattern having a special structure toappear on the surface of an object to be measured. Various patternscaused to appear through the structured light are present, but includeas an example, a plurality of dot patterns arrayed in a lattice shape,stripe-shaped patterns disposed at equal intervals, a lattice pattern,and the like. In addition, the structured light may include not onlysingle-color light, but also multi-color (such as, for example, red,green and blue) light.

The structured light method is a known technique, and thus a detaileddescription thereof will not be given, but the structured light formedby the reflection region provided inside the convex lens 114 causes apattern formed by a plurality of infrared light dots to appear in aregion including the cornea 302 of the user 300.

FIGS. 3(a) and 3(b) are schematic diagrams illustrating reflectionregions 120. FIG. 3(a) is a diagram illustrating when the convex lens114 is seen from the lateral side (outer canthus of the user 300), andFIG. 3(b) is a diagram illustrating when the convex lens 114 is seenfrom the upper side (top of the head of the user 300).

As shown in FIG. 3(a), the near-infrared light source 103 includes aplurality of LEDs 104. To avoid becoming complicated, in FIG. 3(a), onlyone reference numeral 104 is shown, but the rectangles of broken linesindicate the LEDs 104. The LED 104 emits infrared light toward theinside of the convex lens 114.

As shown in FIG. 3(b), a plurality of reflection regions 120 areprovided inside the convex lens 114. To avoid becoming complicated, inFIG. 3(b), only one reference numeral 120 is shown, but regions shown bydiagonal segments in the drawing indicate the reflection regions 120.

As described above, the reflection region 120 is a region having adifferent refractive index as compared to other regions in the convexlens 114. For this reason, the infrared light incident from the LED 104is totally reflected from the reflection region 120 and directed to thecornea 302 of the user 300. Since the reflection region 120 is providedin a plurality of places in the convex lens 114, as much infrared lightas the reflection region 120 is directed to the cornea 302 of the user300. Thereby, dot patterns according to an installation shape of thereflection region 120 can be formed on the cornea 302 of the user 300.Meanwhile, providing a region having a refractive index in the convexlens 114 can be realized using a known laser machining technique.

As described above, the infrared light reaching the cornea 302 of theuser 300 is reflected from the cornea 302 of the user 300, and directedto the direction of the convex lens 114 again. In this case, when theinfrared light reaches the reflection region 120, the infrared light isreflected by the reflection region 120 and is not able to pass throughthe convex lens 114. However, each of the reflection regions 120 is anarrow region, and a relative position between the reflection region 120and the cornea 302 of the user 300 continually changes with a change inthe eyesight of the user 300. For this reason, the probability of theinfrared light reflected from the cornea 302 of the user 300 anddirected to the convex lens 114 being reflected by the reflection region120 is small, which does not lead to a problem.

Even when it is assumed that the infrared light reflected from thecornea 302 of the user 300 and directed to the convex lens 114 isreflected in the reflection region 120 at a certain timing, the relativeposition between the reflection region 120 and the cornea 302 of theuser 300 changes at another timing, and thus the infrared light is notreflected. Therefore, even when reflected light from the cornea 302 ofthe user 300 is reflected by the reflection region 120 at a certainmoment by capturing the infrared light in the camera 116 over time, thecamera 116 can capture an image at another moment, which does not leadto a problem.

FIG. 3(b) illustrates the reflection regions 120 present in a certainhorizontal cross section of the convex lens 114. The reflection regions120 are also present on other horizontal cross sections of the convexlens 114. Therefore, the infrared light emitted from the near-infraredlight source 103 and reflected by the reflection region 120 isdistributed two-dimensionally in the cornea 302 of the user 300 andforms dot patterns.

FIGS. 4(a) and 4(b) are diagrams schematically illustrating an exampleof dot patterns generated by the reflection region 120 of the convexlens 114. More specifically, FIG. 4(a) is a schematic diagramillustrating dot patterns emitted from the convex lens 114. Meanwhile,in the drawing shown in FIG. 4(a), A to H are symbols shown forconvenience to describe the lineup of dots, and are not dot patternscaused to appear in reality.

In FIG. 4(a), black circles and white circles are lined up in alongitudinal direction at equal intervals. Meanwhile, the longitudinaldirection in FIG. 4 is a direction linking the top of the head of theuser to the chin, and is a vertical direction when the user standsupright. Hereinafter, a dot row longitudinally lined up from the symbolA may be described as a row A. The same is true of the symbols B to H.

In FIG. 4(a), the black circle indicates that a dot (bright point)caused by infrared light appears, and the white circle indicates that adot does not appear in reality. In the convex lens 114, a reflectionregion is provided so that a different dot pattern appears at adifferent position of the cornea 302 of the user 300. In an exampleshown in FIG. 4(a), all the dot rows are formed as different dotpatterns from the row A to the row H. Therefore, the cornea coordinateacquisition unit can uniquely specify each dot row by analyzing dotpatterns in an image captured by the camera 116.

FIG. 4(b) is a schematic diagram illustrating dot patterns reflectedfrom a region including the cornea 302 of the user 300, and is a diagramschematically illustrating an image captured by the camera 116. As shownin FIG. 4(b), the dot patterns captured by the camera 116 are dotpatterns mainly reaching the ocular surface of the user 300, and dotpatterns reaching the skin of the user 300 have a tendency to becaptured. This is because the dot patterns reaching the skin of the user300 are diffusely reflected from the skin, and the amount of lightreaching the camera 116 is reduced. On the other hand, dot patternsreaching the ocular surface of the user 300 are subject to reflectionclose to specular reflection due to the influence of tears or the like.For this reason, the amount of light of the dot patterns reaching thecamera 116 also increases.

FIG. 5 is a schematic diagram illustrating a relationship between acaptured dot pattern and a structure of a subject. An example shown inFIG. 5 shows a state where the row D in FIG. 4(a) is captured.

As shown in FIG. 5, the camera 116 captures the cornea 302 of the user300 from a downward direction (that is, direction of the user's mouth).Generally, the human cornea has a shape protruding in an eyesightdirection. For this reason, even when equally-spaced dot patterns appearin the cornea 302 of the user 300, an interval between each of the dotsin the dot pattern captured by the camera 116 changes depending on theshape of the cornea 302 of the user 300. In other words, an intervalbetween the dot patterns appearing in the cornea 302 of the user 300reflects information in a depth direction (that is, direction in whichthe infrared light is emitted to the cornea 302 of the user 300). Thisinterval between the dot patterns is analyzed, and thus the corneacoordinate acquisition unit can acquire the shape of the cornea 302 ofthe user 300. Meanwhile, the above is not limited to when the camera 116captures the cornea 302 of the user 300 from the downward direction.Light paths of infrared light incident on the cornea 302 of the user 300and infrared light reflected from the cornea 302 of the user 300 mayshift from each other, and the camera 116 may capture, for example, thecornea 302 of the user 300 from a transverse direction or an upwarddirection.

FIG. 6 is a diagram schematically illustrating a functionalconfiguration of the image reproducing device 200. The image reproducingdevice 200 includes a reception and transmission unit 210, an imagegeneration unit 220, an eyesight detection unit 230, and a corneacoordinate acquisition unit 240.

FIG. 6 illustrates a functional configuration to operate an imagegeneration process, an eyesight detection process, and a corneacoordinate detection process performed by the image reproducing device200, and other configurations are omitted. In FIG. 6, each componentdescribed as functional blocks that perform various processes can beconstituted by a CPU (Central Processing Unit), a main memory, and otherLSIs (Large Scale Integrations) in a hardware manner. In addition, eachcomponent is realized by programs or the like loaded into the mainmemory in a software manner. Meanwhile, the programs may be stored in acomputer readable recording medium, and may be downloaded from a networkthrough a communication line. It is understood by those skilled in theart that these functional blocks can be realized in various forms byhardware only, software only, or a combination thereof, and are notlimited to any particular one.

The reception and transmission unit 210 executes the transmission ofinformation between the image reproducing device 200 and the headmounted display 100. The reception and transmission unit 210 can berealized by a wireless communication module according to the standard ofMIRACAST (Trademark), WIGIG (Trademark), WHDI (Trademark), or the likedescribed above.

The image generation unit 220 generates an image displayed on the imagedisplay element 108 of the head mounted display 100. The imagegeneration unit 220 can be realized using, for example, the GPU or theCPU described above.

The cornea coordinate acquisition unit 240 analyzes the interval betweenthe dot patterns appearing in the cornea 302 of the user 300, and thusacquires a three-dimensional shape of the cornea 302 of the user 300.Thereby, the cornea coordinate acquisition unit 240 can also estimateposition coordinates of the cornea 302 of the user 300 in athree-dimensional coordinate system using the camera 116 as an origin.

Meanwhile, the camera 116 may be a monocular camera, and may be a stereocamera including two or more imaging units. In this case, the corneacoordinate acquisition unit 240 analyzes the parallax image of thecornea 302 of the user 300 which is captured by the camera 116, and thuscan more accurately estimate the position coordinates of the cornea 302of the user 300 in the three-dimensional coordinate system using thecamera 116 as an origin.

FIG. 7 is a schematic diagram illustrating an eyesight direction of theuser 300. As described above, the dot patterns appearing in the cornea302 are analyzed, and thus the cornea coordinate acquisition unit 240can acquire the shape of the cornea 302 of the user 300. Thereby, asshown in FIG. 6, the eyesight detection unit 230 can detect a peak P ofthe cornea 302 of the user 300 having an approximately hemisphere shape.Subsequently, the eyesight detection unit 230 sets a plane 304 thatcomes into contact with the cornea 302 at the peak P. In this case, thedirection of a normal line 306 of the plane 304 at the peak P is set tothe eyesight direction of the user 300.

Meanwhile, the cornea 302 of the user 300 is generally aspherical ratherthan spherical. For this reason, in the above method in which the cornea302 of the user 300 is assumed to be spherical, an estimation error mayoccur in the eyesight direction of the user 300. Consequently, theeyesight detection unit 230 may provide calibration for an eyesightdirection estimation in advance of the user 300 starting to use the headmounted display 100.

FIG. 8 is a schematic diagram illustrating calibration in an eyesightdirection executed by the eyesight detection unit 230. The eyesightdetection unit 230 generates nine points from points Q.sub.1 to Q.sub.9as shown in FIG. 8 in the image generation unit 220, and displays thesepoints on the image display element 108. The eyesight detection unit 230causes the user 300 to keep observation on these points in order fromthe point Q.sub.1 to the point Q.sub.9, and detects the aforementionednormal line 306. In addition, when the user 300 keeps observation on,for example, the point Q1, the central coordinates (that is, coordinatesof the peak P described above with reference to FIG. 7) of the cornea302 of the user 300 are set to P.sub.1. In this case, the eyesightdirection of the user 300 is set to a direction P.sub.1-Q.sub.1 linkingthe point P1 to the point Q.sub.1 in FIG. 8. The eyesight detection unit230 compares the acquired direction of the normal line 306 with thedirection P.sub.1-Q.sub.1, and stores the error thereof.

Hereinafter, similarly, the user 300 stores errors with respect to ninedirections P.sub.1-Q.sub.1, P.sub.2-Q.sub.2, . . . , P.sub.9-Q.sub.9 ofthe point Q.sub.1 to the point Q.sub.9, and thus the eyesight detectionunit 230 can acquire a correction table to correct the direction of thenormal line 306 obtained by calculation. The eyesight detection unit 230acquires the correction table in advance through calibration, andcorrects the direction of the normal line 306 obtained in theaforementioned method, thereby allowing higher-accuracy eyesightdirection detection to be realized.

It is also considered that, after the user 300 mounts the head mounteddisplay 100 on the head and the eyesight detection unit 230 performscalibration, a relative positional relationship between the head of theuser 300 and the head mounted display 100 changes. However, when theeyesight direction of the user 300 is detected from the shape of thecornea 302 of the user 300 described above, the relative positionalrelationship between the head of the user 300 and the head mounteddisplay 100 does not influence the accuracy of detection of the eyesightdirection. Therefore, it is possible to realize robust eyesightdirection detection with respect to a change in the relative positionalrelationship between the head of the user 300 and the head mounteddisplay 100.

Regarding the above, a method in which the eyesight detection unit 230detects the eyesight direction of the user 300 using a geometric methodhas been described. The eyesight detection unit 230 may execute eyesightdirection detection based on an algebraic method using coordinatetransformation described below, instead of the geometric method.

In FIG. 8, the coordinates of the point Q.sub.1 to the point Q.sub.9 inthe two-dimensional coordinate system which is set in a moving imagedisplayed by the image display element 108 are set to Q.sub.1(x.sub.1,y.sub.1).sup.T, Q.sub.2(x.sub.2, y.sub.2).sup.T . . . , Q.sub.9(x.sub.9,x.sub.9).sup.T, respectively. In addition, the coordinates of theposition coordinates P.sub.1 to P.sub.9 of the cornea 302 of the user300 when the user 300 keeps observation on the point Q.sub.1 to thepoint Q.sub.9 are set to P.sub.1(X.sub.1, Y.sub.1, Z.sub.1).sup.T,P.sub.2(X.sub.2, Y.sub.2, Z.sub.2).sup.T, . . . , P.sub.9(Z.sub.9,Y.sub.9, Z.sub.9).sup.T, respectively. Meanwhile, T indicates thetransposition of a vector or a matrix.

A matrix M having a size of 3.times.2 is defined as Expression (1).

$\begin{matrix}\lbrack {{Math}.\mspace{11mu} 1} \rbrack & \; \\{M = \begin{pmatrix}m_{11} & m_{12} & m_{13} \\m_{21} & m_{22} & m_{23}\end{pmatrix}} & (1)\end{matrix}$

In this case, when the matrix M satisfies Expression (2), the matrix Mbecomes a matrix to project the eyesight direction of the user 300 ontoa moving image surface displayed by the image display element 108.

P.sub.N=MQ.sub.N (N=1, . . . ,9)  (2)

When Expression (2) is specifically written, Expression (3) isestablished.

$\begin{matrix}\lbrack {{Math}.\mspace{11mu} 2} \rbrack & \; \\{\begin{pmatrix}x_{1} & x_{2} & \ldots & x_{9} \\y_{1} & y_{2} & \ldots & y_{9}\end{pmatrix} = {\begin{pmatrix}m_{11} & m_{12} & m_{13} \\m_{21} & m_{22} & m_{23}\end{pmatrix}\begin{pmatrix}X_{1} & X_{2} & \ldots & X_{9} \\Y_{1} & Y_{2} & \ldots & Y_{9} \\Z_{1} & Z_{2} & \ldots & Z_{9}\end{pmatrix}}} & (3)\end{matrix}$

When Expression (3) is deformed, Expression (4) is obtained.

$\begin{matrix}\lbrack {{Math}.\mspace{11mu} 3} \rbrack & \; \\{\begin{pmatrix}x_{1} \\x_{2} \\\vdots \\x_{9} \\y_{1} \\y_{2} \\\vdots \\y_{9}\end{pmatrix} = {\begin{pmatrix}X_{1} & Y_{1} & Z_{1} & 0 & 0 & 0 \\X_{2} & Y_{2} & Z_{2} & 0 & 0 & 0 \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\X_{9} & Y_{9} & Z_{9} & 0 & 0 & 0 \\0 & 0 & 0 & X_{1} & Y_{1} & Z_{1} \\0 & 0 & 0 & X_{2} & Y_{2} & Z_{2} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\0 & 0 & 0 & X_{9} & Y_{9} & Z_{9}\end{pmatrix}\begin{pmatrix}m_{11} \\m_{12} \\m_{13} \\m_{21} \\m_{22} \\m_{23}\end{pmatrix}}} & (4) \\\lbrack {{Math}.\mspace{11mu} 4} \rbrack & \; \\{{y = \begin{pmatrix}x_{1} \\x_{2} \\\vdots \\x_{9} \\y_{1} \\y_{2} \\\vdots \\y_{9}\end{pmatrix}},{A = \begin{pmatrix}X_{1} & Y_{1} & Z_{1} & 0 & 0 & 0 \\X_{2} & Y_{2} & Z_{2} & 0 & 0 & 0 \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\X_{9} & Y_{9} & Z_{9} & 0 & 0 & 0 \\0 & 0 & 0 & X_{1} & Y_{1} & Z_{1} \\0 & 0 & 0 & X_{2} & Y_{2} & Z_{2} \\\vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\0 & 0 & 0 & X_{9} & Y_{9} & Z_{9}\end{pmatrix}},{x = \begin{pmatrix}m_{11} \\m_{12} \\m_{13} \\m_{21} \\m_{22} \\m_{23}\end{pmatrix}}} & \;\end{matrix}$

When the following expression is set, Expression (5) is obtained.

Y=Ax  (5)

In Expression (5), the elements of a vector y are the coordinates of thepoints Q.sub.1 to Q.sub.9 displayed on the image display element 108 bythe eyesight detection unit 230, and thus the elements are known. Inaddition, the elements of a matrix A are coordinates of the peak P ofthe cornea 302 of the user 300 acquired by the cornea coordinateacquisition unit 240. Therefore, the eyesight detection unit 230 canacquire the vector y and the matrix A. Meanwhile, a vector x which is avector obtained by arranging the elements of the transformation matrix Mis unknown. Therefore, when the vector y and the matrix A are known, aproblem of estimating the matrix M becomes a problem of obtaining theunknown vector x.

In Expression (5), when the number of expressions (that is, the numberof points Q presented to the user 300 when the eyesight detection unit230 performs calibration) is larger than the number of unknowns (thatis, the number of elements of the vector x is 6), a prioritydetermination problem occurs. In the example shown in Expression (5),the number of expressions is nine, which leads to a prioritydetermination problem.

An error vector between the vector y and a vector Ax is set to a vectore. That is, the relation of e=y−Ax is established. In this case, in themeaning of minimizing a square sum of the elements of the vector e, anoptimum vector x.sub.opt is obtained by Expression (6).

x.sub.opt=(A.sup.TA).sup.−1A.sup.Ty (6) wherein “−1” indicates aninverse matrix.

The eyesight detection unit 230 constitutes the matrix M of Expression(1) by using the elements of the obtained vector x.sub.opt. Thereby, theeyesight detection unit 230 uses the matrix M and the coordinates of thepeak P of the cornea 302 of the user 300 acquired by the corneacoordinate acquisition unit 240, and thus can estimate where on themoving image surface displayed by the image display element 108 the user300 keeps observation on according to Expression (2).

It is also considered that, after the user 300 mounts the head mounteddisplay 100 on the head and the eyesight detection unit 230 performscalibration, a relative positional relationship between the head of theuser 300 and the head mounted display 100 changes. However, the positioncoordinates of the peak P of the cornea 302 constituting the matrix Adescribed above are values estimated by the cornea coordinateacquisition unit 240 as position coordinates in the three-dimensionalcoordinate system using the camera 116 as an origin. Even when it isassumed that the relative positional relationship between the head ofthe user 300 and the head mounted display 100 changes, a coordinatesystem based on the position coordinates of the peak P of the cornea 302does not change. Therefore, even when the relative positionalrelationship between the head of the user 300 and the head mounteddisplay 100 changes slightly, coordinate transformation according toExpression (2) is considered to be effective. Consequently, eyesightdetection executed by the eyesight detection unit 230 can improverobustness with respect to a shift of the head mounted display 100during mounting.

As described above, according to the head mounted display 100, it ispossible to detect geometric information of the cornea 302 of the user300 wearing the head mounted display 100.

Particularly, the head mounted display 100 can acquire thethree-dimensional shape and the position coordinates of the cornea 302of the user 300, it is possible to estimate the eyesight direction ofthe user 300 with good accuracy.

As stated above, our displays have been described on the basis of ourexamples. The examples have been described for exemplary purposes only,and it can be readily understood by those skilled in the art thatvarious modifications may be made by a combination of each of thesecomponents or processes, which are also encompassed by the scope of thisdisclosure.

In the above, a description has been given of an example when the convexlens 114 is provided with the reflection regions 120 so that differentdot patterns appear at different positions of the cornea 302 of the user300. Dots having different blinking patterns may be caused to appear atdifferent positions of the cornea 302 of the user 300, instead thereofor in addition thereto. This can be realized by forming, for example,the near-infrared light source 103 by a plurality of different lightsources, and changing a blinking pattern in each light source.

Further, although the near-infrared light is radiated from thenear-infrared light source 103 in the above-described embodiment, alight source that radiates near-infrared light may be included in eachpixel constituting the image display element 108. That is, generally,one pixel is constituted by RGB, and a light emitting element that emitsnear-infrared light is provided in addition to the light emittingelements that emit red light, green light, and blue light. When asub-pixel that emits near-infrared light is included as a sub-pixel inthe image display element, the near-infrared light source 103 may not beprovided in the head-mounted display 100.

FIG. 9 is a diagram illustrating a configuration example of the screendisplay element 108. The image display element 108 includes pixelsconstituting an image and sub-pixels constituting the pixels. One pixelwill be described by way of example. A pixel 900 that is one pixelconstituting the image display element 108 includes sub-pixels 900 r,9009, 900 b, and 900 i. The sub-pixel 900 r is a pixel that emits redlight (including light emission of a backlight, light emission of thepixel itself, or light emission of both), the sub-pixel 900 g is a pixelthat emits green light, and the sub-pixel 900 b is a pixel that emitsblue light. In the case of a normal display device, sub-pixels includingthree colors are set as one pixel, or sub-pixels with white light addedthereto are set as one pixel. However, in the case of the screen displayelement 108 according to the present invention, the sub-pixel 900 i isincluded as a sub-pixel.

The sub-pixel 900 i is a pixel that emits near-infrared light. It isdetermined whether or not the sub-pixel 900 i of each pixel 900 emitsthe near-infrared light according to an instruction from the videooutput unit 224, and information indicating whether or not the sub-pixel900 i of each pixel 900 emits the near-infrared light is included indisplay image data that the video output unit 224 outputs to thehead-mounted display 100. Thus, an emission pattern of the near-infraredlight desired by an operator can be formed. Therefore, in a video to bedisplayed at that time, for example, formation of a pattern can also berealized such that the near-infrared light is not emitted according tocontent of the image in a pixel that strongly emits red light. The lightemission of the sub-pixel 900 i may be executed by a display unitincluded in the head-mounted display shown in the above embodiment, ormay be executed by an irradiation unit that controls the sub-pixel 900 ithat emits near-infrared light.

A configuration for emitting near-infrared light in the image iseffective regardless of a type of the display device, and can be appliedto various display devices such as an LCD, a plasma display, an organicEL display. Further, even when a sub-pixel for the near-infrared lightis included in the pixel, the user do not feel uncomfortable whenviewing the image by setting a wavelength of the near-infrared light tobe radiated to be outside a range of wavelengths that can be perceivedby a person with respect to an actually displayed image.

Further, control of the sub-pixel 900 i of which of the respectivepixels of the image display element 108 is caused to emit light may beexecuted by the display unit or the irradiation unit of the head-mounteddisplay 100, or may be executed by the display unit or the irradiationunit of the head-mounted display according to designation of the videogeneration unit 220. Thus, structural light shown in the aboveembodiment can be realized. Further, at this time, turn-on of thesub-pixel 900 i may be appropriately changed. Particularly, for example,when a moving image is displayed on the image display element 108, thesub-pixel 900 i that emits the near-infrared light may be changed eachtime a predetermined time elapses in time series. Here, thepredetermined time may be defined by the number of seconds. In the caseof a moving image, the predetermined time may be defined by the numberof frames, or the predetermined time may be defined for each blankingperiod. In this case, the frame number of the moving image andcoordinate position information of the image display element 108 of thesub-pixel 900 i that emits near-infrared light at that time are storedin the head-mounted display or a line-of-sight detection device inassociation with each other, such that a line-of-sight detection can beappropriately executed each time. Further, a blinking pattern of thesub-pixel 900 i that emits the near-infrared light may be changed in apredetermined period.

Further, a timing at which the sub-pixel 900 i is turned on and a timingat which the sub-pixel 900 r, the sub-pixel 900 g, and the sub-pixel 900b are turned on may be different timings. The camera 116 may beconfigured to execute imaging only at a timing at which the sub-pixel900 i is turned on. Further, as a scheme for realizing thisconfiguration, for example, the configuration may be realized so that ablanking period of the sub-pixel 900 r, the sub-pixel 900 g, and thesub-pixel 900 b and a blanking period of the sub-pixel 900 i are set tobe different time zones. More specifically, it is preferable for theblanking period of the sub-pixel 900 r, the sub-pixel 900 g, and thesub-pixel 900 b to be set as a turn-on period of the sub-pixel 900 i andfor the blanking period of the sub-pixel 900 i to be set as the turn-onperiod of the sub-pixel 900 r, the sub-pixel 900 g, and the sub-pixel900 b.

Further, in the above embodiment, the image is displayed on the imagedisplay element 108 provided on the head-mounted display 100 and thevideo is provided to the user, but the present invention is not limitedthereto. FIG. 10 illustrates an example in which the video is providedto the user without being displayed on the image display element 108that may be adopted by the head-mounted display 100.

A display system 1000 illustrated in FIG. 10 is a display system in acase in which the image displayed on the image display element 108 isnot caused to be visually recognized by the user, but the video isdirectly projected onto the eyes of the user. In recent years, thedevelopment of a virtual retinal display that directly projects thevideo on the eyes of the user has been remarkable. Image light can bedirectly projected without causing any adverse effect on the eyes of theuser, such that the image can be caused to be recognized by the user.This technology can also be applied to the head-mounted display 100according to the present invention. As illustrated in FIG. 10, thedisplay system 1000 directly projects image data transmitted from thevideo output unit 224 onto the eyes 303 of the user via a convex lens114 using an optical fiber 1001. In this case, an invisible light sourceis also included in a pattern as shown in the above embodiment in theimage 1102 and is also radiated on the eyes 303 of the user. Therefore,line-of-sight detection can be realized by imaging a cornea thatreflects the invisible light radiated on the eyes of the user using thecamera 116. Although an example in which the image is directly projectedfrom the optical fiber 1001 to the eyes of the user is shown in FIG. 10,the image light from the optical fiber 1001 may be reflected by a hotmirror or the like and projected onto the eyes 303 of the user via theconvex lens 114.

In the above embodiment, the example in which the line-of-sightdetection is assumed, a marker image is displayed, the marker image iscaused to be gazed by the user, mapping information indicating arelationship between the cornea and a monitor obtained by calibration isstored, and the line-of-sight detection for the user when the user viewsan actual video is performed is shown. However, it goes without sayingthat a line-of-sight detection scheme is not limited to the abovealgorithm. Line-of-sight detection using the following scheme is alsoincluded in the idea of the present invention.

FIG. 11(a) and FIG. 11(b) illustrate an example of a line-of-sightdetection method. A line-of-sight detection unit 230 stores a positionof a corneal center of the user when the user views a center of theimage. FIG. 11(a) is a diagram illustrating an image 1100 obtained byimaging a visible light image including a left eye of the user. Here,although the left eye is used, the same applies to a right eye. It isgenerally known that a landscape that the user views is reflected in theeyes of the user. As illustrated in FIG. 11(a), an image 1110 displayedon the image display element 108 illustrated in FIG. 11(b) is reflectedin the eyes of the user.

When line-of-sight detection using an image reflected in the eyes isrealized, a visible light camera is used as the camera 116. Accordingly,an image based on normal visible light can be imaged, and an image asillustrated in FIG. 11(a) can be acquired. The line-of-sight detectionunit 230 specifies a feature point by performing image analysis such asedge analysis and corner analysis on the obtained image. In FIG. 11(a),for example, feature points 1101 a, 1101 b, and 1101 c can be detected.The line-of-sight detection unit 230 compares the detected feature pointwith a position of the corneal center of the user, specifies the amountof movement from the position of the corneal center of the user when theuser views the center of the image stored in advance, and detects aplace (line-of-sight direction) at which the user is gazing. Such aconfiguration may be used. When the line-of-sight detection using theimage reflected in the eyes of the user is performed, the head-mounteddisplay 100 needs to include a half mirror that partly reflects visiblelight and partially transmits the visible light in place of the hotmirror 112 or needs to include a visible light camera that directlyimages the eyes of the user separately from the camera 116.

Further, although the position of the corneal center of the user whenthe user is viewing the center of the image is stored in the abovedescription, the line-of-sight detection can be performed withoutstoring the position information. That is, the feature point is detectedfrom a first frame of the moving image output by the video output unit224 and a second frame following the first frame (the second frame maynot be a frame immediately after the first frame, but at least a part ofthe same object as an object to be displayed in the first frame isrequired to be displayed). Further, a position of a corneal center ofthe user gazing at the first frame at that time and a position of acorneal center of the user gazing at the second frame are detected. Theline-of-sight detection unit 230 may be configured to detect a point(line-of-sight direction) of the second frame at which the user isgazing on the basis of a movement distance and a movement direction onthe screen display element 108 from the feature point in the first frameto the corresponding feature point in the second frame, and a movementdistance and a movement direction on the screen display element 108 fromthe position of the corneal center of the user in the first frame to theposition of the corneal center of the user in the second frame.

According to these schemes, it is not necessary to execute thecalibration by displaying the marker image shown in the aboveembodiment. Therefore, prior preparation for performing theline-of-sight detection using the head-mounted display 100 may not beperformed, and convenience of the user can be improved.

INDUSTRIAL APPLICABILITY

The present invention is applicable to a head mounted display.

REFERENCE SIGNS LIST

-   -   1 Image system    -   100 Head mounted display    -   103 near-infrared light source    -   104 LED    -   108 Image display element    -   112 Hot mirror    -   114 Convex lens    -   116 Camera    -   118 Image output unit    -   120 Reflection region    -   130 Image display system    -   150 Housing    -   160 Mounting fixture    -   170 Headphone    -   200 Image reproducing device    -   210 Reception and transmission unit    -   220 Image generation unit    -   230 Eyesight detection unit

1. A line-of-sight detection system comprising a head-mounted displayand a line-of-sight detection device, the head-mounted display includesan image display element that includes a plurality of pixels, each pixelincluding sub-pixels that emit red, green, blue, invisible light, anddisplays an image to be viewed by a user; an imaging unit that images aneye of the user wearing the head-mounted display on the basis of theinvisible light emitted from the sub-pixel that emits the invisiblelight; and a transmission unit that transmits a captured image capturedby the imaging unit, and the line-of-sight detection device includes areception unit that receives the captured image; and a line-of-sightdetection unit that detects a line of sight of the eye of the user basedon the captured image.
 2. The line-of-sight detection system accordingto claim 1, wherein the head-mounted display further include a controlunit that selects the pixel that emits invisible light among theplurality of pixels constituting the image display element and causesthe selected pixel to emit light.
 3. The line-of-sight detection systemaccording to claim 2, wherein the control unit changes the pixel thatemits the invisible light when a predetermined time elapses.
 4. Theline-of-sight detection system according to claim 2, wherein the controlunit switches and controls a light emission timing of a sub-pixel thatemits the invisible light and the sub-pixel other than the sub-pixelthat emits the invisible light.
 5. A head-mounted display mounted on ahead of a user and used, the head-mounted display comprising: a convexlens disposed in a position facing a cornea of the user when thehead-mounted display is mounted; an image display element that includesa plurality of pixels, each pixel including sub-pixels that emit red,green, blue, invisible light, and displays an image to be viewed by auser; a camera that images a video including the cornea of the user as asubject; and a housing that houses the convex lens, the image displayelement, and the camera.
 6. The head-mounted display according to claim5, further comprising a control unit that selects the pixel that emitsinvisible light among the plurality of pixels constituting the imagedisplay element and causes the selected pixel to emit light.
 7. Thehead-mounted display according to claim 6, wherein the control unitchanges the pixel that emits the invisible light when a predeterminedtime elapses.
 8. The head-mounted display according to claim 6, whereinthe control unit switches and controls a light emission timing of asub-pixel that emits the invisible light and the sub-pixel other thanthe sub-pixel that emits the invisible light.
 9. A line-of-sightdetection method using a line-of-sight detection system comprising ahead-mounted display including an image display element that includes aplurality of pixels, each pixel including sub-pixels that emit red,green, blue, invisible light, and displays an image to be viewed by auser, and a line-of-sight detection device, the line-of-sight detectionmethod comprising: an irradiation step of emitting, by the head-mounteddisplay, invisible light from sub-pixels of the plurality of pixels andirradiating a cornea of the user with the invisible light; an imagingstep of imaging, by the head-mounted display, a subject radiated withinvisible light emitted from the sub-pixel that emits the invisiblelight and including a cornea of the user viewing the image displayed onthe image display element to generate a captured image; a transmissionstep of transmitting, by the head-mounted display, the captured image tothe line-of-sight detection device; a reception step of receiving thecaptured image by the line-of-sight detection device; and a detectionstep of detecting, by the line-of-sight detection device, a direction ofa line of sight of the user on the basis of the captured image.
 10. Aline-of-sight detection system comprising a head-mounted display and aline-of-sight detection device, wherein the head-mounted displayincludes an image display element that displays an image to be viewed bya user; an imaging unit that images an eye of the user in which theimage displayed on the image display element is reflected; atransmission unit that transmits a captured image captured by theimaging unit, and the line-of-sight detection device includes areception unit that receives the captured image; and a line-of-sightdetection unit that detects a direction of a line of sight of the eye ofthe user on the basis of a feature point of the image displayed on theimage display element included in the captured image and a feature pointof the image displayed on the image display element.